Repository navigation
[bugfix]: fix VAE temporal tiling blend corruption in tiled_encode - #1181
Conversation
HunyuanVideo is ~13B params and OOMs with replicate-only FSDP. Switch to hsdp_shard_dim=8 to shard across all GPUs. Add smoke test config that uses Wan data (shapes match) for quick iteration.
…ript - Replace smoke test config with overfitting config (500 steps, higher LR, validation every 100 steps, precondition_outputs) - Add preprocess_hunyuan_overfit.py for encoding small video sets with HunyuanVideo VAE + LLaMA + CLIP into parquet format
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request resolves a critical bug in the VAE's temporal tiling encoding process that was causing severe corruption and quality loss in generated videos. By ensuring that temporal blending in latent space correctly accounts for the Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new configuration for HunyuanVideo T2V overfitting and a new preprocessing script for this data. It also includes a change in the VAE's tiled_encode method to adjust blending frames based on temporal compression. Review comments suggest moving imports to the top of the preprocessing script for better organization and removing an unused variable (num_latent_t) for clarity.
| from transformers import AutoTokenizer, CLIPTextModel, LlamaModel | ||
| from fastvideo.configs.pipelines.hunyuan import ( | ||
| HunyuanConfig, LlamaConfig, CLIPTextConfig) |
There was a problem hiding this comment.
For improved code organization and to adhere to standard Python style conventions (PEP 8), all imports should be placed at the top of the file. This makes dependencies clear and avoids potential issues with delayed imports. Please move these transformers and fastvideo imports to the top-level of the script.
| # Compute latent frames | ||
| num_latent_t = latent.shape[1] |
There was a problem hiding this comment.
The variable num_latent_t is assigned but never used. The accompanying comment is also slightly misleading. To improve code clarity, this unused variable and its comment should be removed.
Pre-commit checks failedHi @alexzms, the pre-commit checks have failed. To fix them locally: # Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install
# Run all checks and auto-fix what's possible
pre-commit run --all-filesCommon fixes:
After fixing, commit and push the changes. The checks will re-run automatically. For future commits, |
Remove unused variable num_latent_t, add type annotations for mypy, and apply yapf formatting.
Summary
tiled_encode()using pixel-spaceblend_num_framesin latent space, causing severe quality loss (PSNR 25.7 → 38.6 dB after fix)Details
blend_num_frameswas computed in pixel space (16 - 12 = 4) but used directly as latent-space blend extent intiled_encode(). With 4x temporal compression, only ~1 latent frame overlaps between tiles, but 4 were blended, thus overwriting unrelated content. Fix: divide bytemporal_compression_ratiowhen blending in latent space. Decode paths are unaffected (they blend in pixel space).Other models vae will not get affected theoretically
Test plan